Managing Multimodal Data, Metadata and Annotations: Challenges and Solutions

The application of statistical learning to multimodal signal processing requires a significant amount of data for development, training and test. The availability of data often conditions the possibility of an investigation, and can influence its objectives and application domains. The development of innovative multimodal signal processing methods depends not only on raw data recorded from various sources, but more specifically on high added-value data that is accompanied by ground-truth metadata or annotations. This information, generally added or validated manually, is used for supervised learning and for testing. The goal of this chapter is to outline the main stages in multimodal data management, starting with the capture of multimodal raw data in instrumented spaces (Section 11.3). The challenges of data annotation – mono or multimodal – are discussed in Section 11.4, while the issues of data formatting, storage and distribution are analyzed in Section 11.5. In particular, Section 11.5.3 provides a discussion of the access to multimodal data sets using interactive tools, and more specifically of some meeting browsers that were recently developed to access multimodal recordings of humans in interaction. The chapter starts, however, by introducing in the following section (11.2) an important conceptual distinction between metadata and annotations, and then sur-